Added simple example script for running multiple run control apps by bieryAtFnal · Pull Request #333 · DUNE-DAQ/daqsystemtest

bieryAtFnal · 2026-08-06T16:40:48Z

Description

When I was starting to think about how we might add support for multiple, user-specified applications to be run as part of our integration tests, I ran some Google searches that produced sample python code that was very helpful.

I modified that sample code to start drunc-unified-shell, drunc-process-manager, and drunc-process-manager-shell processes.

The end result was so helpful that I wanted to capture the result for later use.

This PR adds this script to this repo.

The script can by run by typing multiprocess_runcontrol_driver.py after including this branch in your software area and re-building the code in that software area.

Once the script has been started, commands can be sent to one of the three processes by pre-pending the process nickname to the command (with a colon separator). For example, drunc:ps.

Typing exit (with no process prefix) will exit the script.

Type of change

New feature or enhancement (non-breaking change which adds functionality)

… unified shell as separate processes.

…_driver.py

…iproc_runcontrol_script

emmuhamm

Okay this is pretty epic, thanks Kurt!

I was able to install and run this just fine locally, being able to spawn this which is basically a 'wrapper' for multiple apps which is very very cool. Everything mostly works as expected.

I've left a few minor comments, but none I would consider blockers. I would prefer this to go in rather than waiting some time to make this perfect since base functionality works and this can always be iterated upon later.

Anyways consider this a soft approval from my side. I can approve this with the button if the comments have been addressed.

emmuhamm · 2026-08-20T09:44:23Z

+    interactive_cmds = [
+        ["pm", "drunc-process-manager", "ssh-standalone", str(pm_port)],  # Launch Interactive Python Instance 1
+        ["pmshell", "drunc-process-manager-shell", f"grpc://localhost:{pm_port}"],  # Launch Interactive Python Instance 2
+        ["drunc", "drunc-unified-shell", f"grpc://localhost:{pm_port}", "config/daqsystemtest/example-configs.data.xml", "local-1x1-config", "biery-local-test"]   # Launch Interactive Python Instance 3
+    ]


Is the purpose of this script for a test/demonstration of the PMaaS functionality or are people expected to use this more generally / in a wider use case?

If its the latter, then maybe people might want to change these lines to fit their needs? I would suggest the following:

The easiest might be a comment / some docs here that says 'please change this to change what set of applications you would like to run'. In these comments, it would have to be clear that the first element of this list is the 'alias' of the application and the rest are just the usual application setups

Something harder, possibly not worth it, is to open this up as an runtime argument. I'm ambivalent / semi against this idea though

emmuhamm · 2026-08-20T09:45:02Z

+                        if "drunc" in target:
+                            proc.stdin.write(("echo '*** COMMAND HAS COMPLETED ***'\n").encode())
+                            await proc.stdin.drain()
+                            #print(f"[System] Sent to {target}: echo '*** COMMAND HAS COMPLETED ***'")


Here and a couple of others: Some stray comments that should be deleted before merge into develop

emmuhamm · 2026-08-20T09:46:31Z

@@ -0,0 +1,140 @@
+#!/bin/env python3


I thikn the top of this file is missing a bit of documentation. People should know how to run this and what this script does, as well as a brief description of how to interact with the shells. (and then maybe add a todo to link these to the PMaaS docs when we eventually make them >.<)

emmuhamm · 2026-08-20T09:49:48Z

+    print("*** Type 'exit' to quit everything.")
+    print()
+
+    # 3. Handle interactive user input from the main terminal


One thing I noticed is that arrow keys don't work. Makes a bit hard to submit the previous commands.

I dont think this is a blocker for this merge tho. It would be nice if it can make it but if its too difficult I'm happy for this to be deferred

emmuhamm · 2026-08-20T09:53:26Z

+            if ":" in command_text:
+                target, msg = command_text.split(":", 1)
+                target = target.strip()
+
+                if target in processes:
+                    proc = processes[target]
+                    if proc.returncode is None:  # Check if still running
+                        cmd_start_time = time.time()
+                        proc.stdin.write((msg + "\n").encode())
+                        await proc.stdin.drain()
+                        print(f"[System] Sent to {target}: {msg}")
+                        if "drunc" in target:
+                            proc.stdin.write(("echo '*** COMMAND HAS COMPLETED ***'\n").encode())
+                            await proc.stdin.drain()
+                            #print(f"[System] Sent to {target}: echo '*** COMMAND HAS COMPLETED ***'")
+                            await command_completion_event.wait()
+                            command_completion_event.clear()
+                        else:
+                            now = time.time()
+                            #print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)
+                            while True:
+                                if last_msg_time <= cmd_start_time:
+                                    if now - cmd_start_time > 5:
+                                        break
+                                else:
+                                    if now - last_msg_time >= 5:
+                                        break
+                                #print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)
+                                await asyncio.sleep(0.25)
+                                now = time.time()
+                            #print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)
+                    else:
+                        print(f"[System] Error: {target} has already exited.")
+                else:
+                    print(f"[System] Error: Process '{target}' not found.")
+            else:
+                print("[System] Invalid format. Use: <process_name>:<command>")


Quite a nested block of code. I prefer if its inverted to flatten them:

Suggested change

if ":" in command_text:

target, msg = command_text.split(":", 1)

target = target.strip()

if target in processes:

proc = processes[target]

if proc.returncode is None: # Check if still running

cmd_start_time = time.time()

proc.stdin.write((msg + "\n").encode())

await proc.stdin.drain()

print(f"[System] Sent to {target}: {msg}")

if "drunc" in target:

proc.stdin.write(("echo '*** COMMAND HAS COMPLETED ***'\n").encode())

await proc.stdin.drain()

#print(f"[System] Sent to {target}: echo '*** COMMAND HAS COMPLETED ***'")

await command_completion_event.wait()

command_completion_event.clear()

else:

now = time.time()

#print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)

while True:

if last_msg_time <= cmd_start_time:

if now - cmd_start_time > 5:

break

else:

if now - last_msg_time >= 5:

break

#print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)

await asyncio.sleep(0.25)

now = time.time()

#print(f"{cmd_start_time} {last_msg_time} {now}", flush=True)

else:

print(f"[System] Error: {target} has already exited.")

else:

print(f"[System] Error: Process '{target}' not found.")

else:

print("[System] Invalid format. Use: <process_name>:<command>")

if ":" not in command_text:

print("[System] Invalid format. Use: <process_name>:<command>")

continue

target, msg = command_text.split(":", 1)

target = target.strip()

if target not in processes:

print(f"[System] Error: Process '{target}' not found.")

continue

proc = processes[target]

if proc.returncode is not None:

print(f"[System] Error: {target} has already exited.")

continue

cmd_start_time = time.time()

proc.stdin.write((msg + "\n").encode())

await proc.stdin.drain()

print(f"[System] Sent to {target}: {msg}")

if "drunc" in target:

proc.stdin.write(("echo '*** COMMAND HAS COMPLETED ***'\n").encode())

await proc.stdin.drain()

await command_completion_event.wait()

command_completion_event.clear()

continue

now = time.time()

while True:

if last_msg_time <= cmd_start_time and now - cmd_start_time > 5:

break

if last_msg_time > cmd_start_time and now - last_msg_time >= 5:

break

await asyncio.sleep(0.25)

now = time.time()

Comments on each if statement as necessary too if theyre not clear enough

…iproc_runcontrol_script

bieryAtFnal · 2026-08-20T14:20:29Z

Thanks, @emmuhamm, for taking a look.
As you probably guessed, my goal in submitting the current state of this script for review was more in the spirit of wanting to capture it so that I/we don't lose track of it or forget about it than in the spirit of it being ready for production or wide use.
So, I appreciate your willingness to let it move ahead now and get modified/improved later.
That said, I want to implement some of your suggestions now (e.g. comments at the top of the script to describe its use). I'll look into those soon.

Kurt Biery added 6 commits July 16, 2026 10:09

Adding AI-assisted script for running run control process manager and…

ca4b56b

… unified shell as separate processes.

added messages for the user to multiprocess_runcontrol_driver.py

92356ce

added pm shell to multiprocess_runcontrol_driver.py

a065491

Added logic to wait for command completion in multiprocess_runcontrol…

4c9bf7c

…_driver.py

Merge remote-tracking branch 'origin/develop' into kbiery/sample_mult…

b111aa4

…iproc_runcontrol_script

Merge remote-tracking branch 'origin/develop' into kbiery/sample_mult…

c76bd6e

…iproc_runcontrol_script

bieryAtFnal marked this pull request as ready for review August 18, 2026 13:25

bieryAtFnal requested a review from emmuhamm August 18, 2026 21:28

emmuhamm reviewed Aug 20, 2026

View reviewed changes

Merge remote-tracking branch 'origin/develop' into kbiery/sample_mult…

be67a3e

…iproc_runcontrol_script

Navigation Menu

Uh oh!

Added simple example script for running multiple run control apps - #333

Added simple example script for running multiple run control apps#333
bieryAtFnal wants to merge 7 commits into
developfrom
kbiery/sample_multiproc_runcontrol_script

bieryAtFnal commented Aug 6, 2026 •

edited

Loading

Uh oh!

emmuhamm left a comment

Uh oh!

emmuhamm Aug 20, 2026

Uh oh!

emmuhamm Aug 20, 2026

Uh oh!

emmuhamm Aug 20, 2026

Uh oh!

emmuhamm Aug 20, 2026

Uh oh!

emmuhamm Aug 20, 2026

Uh oh!

bieryAtFnal commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bieryAtFnal commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Uh oh!

emmuhamm left a comment

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

emmuhamm Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

bieryAtFnal commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bieryAtFnal commented Aug 6, 2026 •

edited

Loading